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Abstract. Teachers and students alike would benefit from self-access materials to free up 
lesson time for activities requiring live face-to-face interaction, but such materials must 
first undergo thorough testing and evaluation of their pedagogical effectiveness. The 
present study is part of an ongoing project to develop a series of self-paced, interactive 
online modules on the use of grammatical voice. Previous acceptability judgment data 
showed these materials to be effective for improving performance on sentences with 
animate subjects but largely ineffective or even detrimental in cases with inanimate 
ones. Thus, 10 returning participants were individually recorded re-attempting the same 
acceptability judgments, this time stimulated by structured interview questions to probe 
decision-making processes and thereby pennit step-by-step analysis of conceptual 
understanding. The findings included common failure to grasp the role of personification 
in permitting false agentive subjects in English, conflation of the use of inanimate subjects 
with passive voice, and unawareness of relationships between grammatical case-marking 
and thematic relations. 
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1. Introduction 

Attainment of the advanced language proficiency necessary for academic and 
professional use generally requires many more hours of study than most programs can 
provide in the classroom (Lyddon, 2011). Exacerbating this problem are seemingly 
insurmountable difficulties such as the teaching of grammatical voice (Hinkel, 2002; 
Master, 1991; Owen, 1993). On this latter issue, Johnson and Lyddon (in preparation) 
have found promising signs of success with a concept-based approach to instruction, 
as advocated by Negueruela (2003). However, as the implementation of their face-to- 
face lessons takes three entire 90-minute class periods, Lyddon (2012a, 2012b) has 
experimented with an online alternative, only to find mixed results in learner outcomes, 


* Contact author: palyddon@kanda.kuis.ac.jp 


In L. Bradley & S. Thouesny (Eds.), CALL: Using, Learning, Knowing, EUROCALL Conference, Gothenburg, Sweden, 
22-25 August 2012, Proceedings (pp. 192-197). © Research-publishing.net Dublin 2012 


192 




A Diagnostic Approach to Improving the Pedagogical Effectiveness of Tutorial CALL Materials 


including inferior performance on acceptability judgments involving sentences 
with inanimate subjects and active verbs. Thus, the current study represents a post¬ 
use evaluation as part of a pedagogy-based approach to tutorial CALL development 
(Colpaert, 2006) in order to not only explain the previous findings but to inform the 
next iteration of the instructional design. 

2. Method 

2.1. Procedures 

The data were collected in a two-hour, one-on-one session with each of 10 participants 
over a period of four weeks in the spring of 2012 at a computer science university in 
northern Japan. The 10 participants were a representative sample of the 31 who had 
participated in the immediately prior pilot study (Lyddon, 2012a). 

Although each data collection session was conducted face to face, with the exception 
of paper-based mark sheets, all interactions between the participants and the researcher 
were mediated by a shared laptop computer and recorded using Camtasia Studio® 
screen capture software. Each session began with a self-paced, 10-frame Adobe® 
Director® movie giving a summary review of the three learning modules from the 
pilot study (i.e., on agency, animacy, and transitivity). Each participant then took a 
self-paced version of the otherwise identical pre-posttest acceptability judgment task 
(i.e., five randomized permutations of a single sentence for each of six regular verbs: 
analyze, damage, design, discuss, occur, and study). The researcher then revisited each 
item with the participant, asking whether the sentence appeared active or passive and 
recording the answer without comment. 

Next, the researcher began a bilingual interview focused on a slide presentation 
featuring a systematic rearrangement of the same items as for the acceptability 
judgment task. First, the items were grouped alphabetically by verb, with the exception 
of ‘occur’, the only pure intransitive, which came last. Within each group, they were 
then ordered according to apparent conceptual difficulty: animate subjects with active 
transitive verbs (AAT), animate subjects with passive transitive verbs (APT), inanimate 
subjects with active intransitive verbs (IAI), inanimate subjects with passive verbs (IP), 
inanimate subjects with active transitive verbs (IAT). 

The interview protocol proceeded with the researcher first showing the participant 
an isolated verb in its dictionary form and asking a short series of questions about 
its perceived properties. The participant was then shown the five permutations of 
the same sentence for the given verb in the order indicated above (i.e., AAT, APT, 
IAI, IP, IAT). For each sentence, a standard sequence of probe questions was used 
to elicit the participant’s considerations in making an acceptability decision. Finally, 
the participant re-took the same self-paced version of the acceptability judgment task 
from the start of the session. 
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2.2. Results 

With only 10 participants, this study was underpowered to find statistical 
significance for any but the largest of effect sizes. Nevertheless, paired-samples 
comparisons between the scores on the original pretest (M = 23.1, SD = 3.04) and 
those on the delayed posttest given just after the review (M = 23.8, SD = 3.12) 
were similar to those on the original posttest (M = 24.2, SD = 2.90), showing 
gains of .36 and .34 standard deviational units, respectively. In other words, the 
nominal decrease on the delayed posttest with respect to the original posttest was 
not important. 

On revisiting each posttest item to identify it as either active or passive, 7 out of 10 
participants named all 30 correctly. Of the items the other three participants missed, 
nearly 95% (17 out of 18) involved inanimate subjects with active verbs, two thirds of 
which were intransitive. Incorrect responses were observed for all verbs but ‘analyze’, 
with ‘study’ also posing no problem for the 1AT sentences. 

The probe later revealed not only confusion of the passive with the use of inanimate 
subjects but also initial conflation of passive voice with past tense. It also became clear 
that many students, although they correctly understood the terms for ‘transitive’ and 
‘intransitive’ in Japanese, had them backwards in English. 

Another finding was that most students seemed unaware of the general need for 
instrumental case marldng in English, accepting sentences like ‘Two main ways 
analyzed the results’ until given a literal Japanese translation with the explicit 
nominative case marker ‘ ga ’ on ‘ houhou ’ (‘ways’). Those who rejected such sentences 
focused exclusively on the literal capabilities of inanimate subjects. Consequently, no 
one correctly identified the acceptable use of the false agentive subject in ‘A previous 
paper discussed the issue’. 

Despite the limited number of participants, a paired-samples t-test at a .05 alpha 
level comparing the scores on the acceptability judgment task before (M = 23.8, SD 
= 3.12) and after (M= 27.4, SD = 2.50) the interview protocol showed a statistically 
significant increase and a large effect: t( 9) = 2.903, p = .02, d = .92. Accounting for 
much of the difference was noticeable improvement on all sentence types for ‘occur’ 
(Table 1) and all verbs except ‘damage’ for IAT sentences (Table 2). 

Table 1. Pre- and post-protocol acceptability judgments on the verb ‘occur’ 


Sentence 

Type 

Students Answering Correctly (%) 

Pre 

Post 

AAT 

20 

90 

APT 

80 

90 

IAI 

50 

100 

IP 

60 

90 

IAT 

30 

90 
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Table 2. Pre- and post-protocol acceptability judgments on IAT sentences 


Verb 

Students Answering Correctly (%) 

Pre 

Post 

analyze 

50 

80 

damage 

80 

70 

design 

50 

80 

discuss 

70 

100 

study 

70 

80 


3. Discussion 

Despite their re-randomized presentation order, the post-interview acceptability 
judgment items were the same as those used in the training. As such, although it is 
doubtful that the learners were able to make the correct judgments on so many 
seemingly similar sentences based simply on short-term memory, it is also unlikely 
that the magnitude of improvement would have been so great with a different set of 
verbs, for the probing procedure made the inadequacy of their lexical knowledge 
clearly evident. Nevertheless, such remarkable gains on even this small number of 
verbs suggest several insights from the interview protocol that might be fruitfully 
implemented in the next stage of the module development. 

First, more attention needs to be given to vocabulary, starting with the essential 
terminology. While one might argue that the teaching of metalinguistic target language 
is premature or even unnecessary, most of the students who participated in this 
research, although they were monetarily compensated, chose to do so primarily for the 
opportunity to interact in English as much as possible. Thus, any initial use of the native 
language must serve the purpose of scaffold as well as explanatory tool. This part of 
the instruction will then need to include the difference between past and passive as well 
as transitive and intransitive and subject and agent. Of course, abstract explanations 
alone may be difficult to grasp, especially since they often contain additional unfamiliar 
words. As such, it may be necessary to use examples involving only animate entities 
(rare in computer science discourse) to illustrate phenomena such as the shifting 
position of the agent in the process of passivization (e.g., The cat killed the mouse vs. 
The cat was killed by the mouse). 

Second, additional sensitization to grammatical form is needed. While learners 
might be excused for confusing the past tense with the past participle, it is clear that they 
were also ignoring the presence or absence of a preceding copula and the subsequent 
‘by’ when deciding agent and patient roles. As it is easier to notice the presence of 
something rather than its absence, this training might be achieved through the 
juxtaposition of similar sentences, which might also serve to reinforce the differences 
between nominative and instrumental cases (e.g., A computer simulation studied the 
effects vs. We studied the effects in a computer simulation). 
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Finally, confirmation probes should be built into the instruction to guide the learners 
as they struggle to master the new concepts. For instance, often a participant would 
correctly identify a verb as transitive but then incorrectly accept a sentence using 
that verb without a direct object. After simply being asked again about transitivity, 
however, he or she usually not only quickly spotted the error but also explained it, thus 
reinforcing conceptual understanding. 

4. Conclusions 

The findings of the current investigation indicate that a number of factors, including 
confusion over linguistic terminology, inattention to grammatical forms, and 
insufficient knowledge of the target verbs, all may have contributed to the mixed results 
of the previous study. As the interview protocol employed to obtain these data led to 
unequivocally superior acceptability judgment performance thereafter, key elements 
from the probing procedure will be incorporated and evaluated in a future iteration of 
the pedagogical design. 
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